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e Configuration (read-only) 
e Runtime state (read-write) 


Connection limits data 
Request limits data 
Proxy cache metadata 
Upstream servers status 
SSL sessions cache 
Stub status counters 


e Dynamically configured upstream servers @ 
e Upstream health-check status & 

e Sticky sessions to upstreams & 

e API module metrics & 

e Key-Value Store 9 

e HTTP Session log data $ 
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Implementation of state sharing 


nginx uses shared memory approach: 
e requires explicit configuration 
+ data access is effective 
+ data is always consistent, but locked access is usually required 
— if a process crashes during an update, data is left in inconsistent state 


Sharing Information within Cluster 


Can we just extend shared memory approach to the network ? 


Sharing Information within Cluster 


Can we just extend shared memory approach to the network ? 
e consistent access to data is expensive and complex: 


Sharing Information within Cluster 


Can we just extend shared memory approach to the network ? 


e consistent access to data is expensive and complex: 
e to response, node must interact with others 


Sharing Information within Cluster 


Can we just extend shared memory approach to the network ? 


e consistent access to data is expensive and complex: 


e to response, node must interact with others 
e hard to implement and configure properly 


Sharing Information within Cluster 


Can we just extend shared memory approach to the network ? 


e consistent access to data is expensive and complex: 


e to response, node must interact with others 
e hard to implement and configure properly 
e network nodes are ephemeral, chances to fail are high 


Sharing Information within Cluster 


Can we just extend shared memory approach to the network ? 


e consistent access to data is expensive and complex: 
e to response, node must interact with others 
e hard to implement and configure properly 
e network nodes are ephemeral, chances to fail are high 


e the result of such efforts will be a distributed database 


Sharing Information within Cluster 


Can we just extend shared memory approach to the network ? 


e consistent access to data is expensive and complex: 
e to response, node must interact with others 
e hard to implement and configure properly 
e network nodes are ephemeral, chances to fail are high 

e the result of such efforts will be a distributed database 


e ... nginx is nota distributed database (and is not going to be)! 
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So what we want from an nginx in a clustered environment? 
e process input on each node separately 
e inform other nodes about local changes 
e process new input with new knowledge 
What useful scenarios we can implement? 
e configuration updates 
e session caching 
e apply resource limiting 


Offered Solutions in Nginx-Plus 


The following modules are available: 
e Sticky sessions synchronization (R15) 


Offered Solutions in Nginx-Plus 


The following modules are available: 
e Sticky sessions synchronization (R15) 
e Key-Value store synchronization (R16) 


Offered Solutions in Nginx-Plus 


The following modules are available: 
e Sticky sessions synchronization (R15) 
e Key-Value store synchronization (R16) 
e Distributed request rate limiting (R16) 
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Nginx-Plus R15: the ngx zone sync Module 


e TCP based stream module 

e Native nginx asynchronous I/O 

e Full mesh 

e Cluster nodes discovery via DNS 

e Very simple push protocol with minimal overhead 

e Content-based synchronization (sync objects, not bytes) 
e Full TLS support 

e Configurable latency/resources balance 
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upstream backend { 
server ...; 
server ...; 


sticky learn create=$ups_key 
lookup=$key 
zone=z: 10m : 


+ 
server { 
location / < 
proxy. pass http://backend; 
+ 
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syncing Sticky Sessions 


http { stream { 
upstream backend { server { 
server ...; zone_sync; 
server ...; listen 192.168.1.1:12345; 
# mark zone ’z’ for synchronization 
sticky learn create=$ups_key # cluster nodes list (including self) 
lookup=$key zone.syne_.server 192.168:1.1:12348; 
zone=z:10m sync; zone_sync_server 192.168.1.2:12345; 
+ À: 
server { } 


location / { 


proxy_pass http://backend; 
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stream { 


server { 
zone_sync; 
listen 192.168.1.1:12345; 


# cluster nodes list (including self) 


zone sync server 192.168.1.1:12546; 
zone sync_server 192.168,1.,2:12345; 
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Using DNS to Obtain Cluster Nodes 


stream { 
resolver 127.0.0.1; 
server { 
zone_sync; 
listen 192.168.1.1:12345; 


zone_sync_server cluster.example.com:12345 resolve; 


http { cee 4 


Secure Connections Between Nodes with SSL 


zone_sync; 
listen 192.168.1.1:12345 ssl; 


ssl_certificate foo.crt; 
ssl_certificate_key foo.key; 


zone_sync_ssl on; 

zone_sync_ssl_certificate node.crt; 
zone_sync_ssl_certificate_key node.key; 
zone_sync_ssl_verify on; 
zone_sync_ssl_trusted_certificate trusted.crt; 


Tuning Synchronization: latency vs bandwidth/CPU 


zone_sync; 


listen ...; 
zone_sync_ 


zone_sync_ 
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zone_sync_ 
zone_sync_ 
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server ...; 
interval 500ms; 
buffers 256 8k; 


recv_buffer_size 8k; 
timeout 5s; 


More modules 


e Key-Value storage (http and stream): 
keyval_zone zone=one:32k state=one.keyval sync; 
e HTTP Request Limiting: 


limit_req_ zone $binary_remote_addr 
zone=one : 10m 
rate=1r/s sync; 


fl i 


Thank you! Links: 


e https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html 
° https://www.nginx.com/blog/nginx-plus-r15-released/#state-sharing 

e hitps://www.nginx. com/blog/nginx- plus-r16-released/#r16- cluster- rate-limiting 
e hitps://www.nginx. com/blog/nginx- Rus -r16- released/#r16-cluster- -key-value 


Questions? 
e Owen Garret / owen@nginx.com 


e Vladimir Khomutov / vl@nginx.com 


